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FOREWORD 


This document defines the Implementation/Operations Plan for the 
evaluation of information pertaining to wheat area, yield, and production 
generated by the Crop Assessment Subsystem (CAS) of the Large Area Crop 
Inventory Experiment (LACIE). This document is valid through Phase III 
of LACIE. 

This is a controlled LACIE document. Changes, additions or deletions 
to this document are controlled by the Level 3 LACIE Change Control Board 
(CCB). Direct all inquiries to R.B. Mac Donald, NASA-JSC, LACIE Project 
Manager. 


Approved by: 






R.B. MacDonald, LACIE Manager 


for J.D. Murphy', Deputy Manager 
Technical Advisor 
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SECTION 1.0 . 
INTRODUCTION 


This document presents the' Implementation/Operations 
Plan for the evaluation of information pertaining to 
wheat area, yield, and production generated "by the 
Large Area Crop Inventory Experiment (LACIE) Appli- 
cation Evaluation System (AES). Information Eval- 
uation (IE) will he performed at the U.S. Department 
of Agriculture (USDA) LACIE Headquarters with prime 
responsibility within the USDA and participation by 
the National Aeronautics and Space Administration 
(NASA) and the National Oceanic and Atmospheric 
Administration (N0AA) o Within this function the 
USDA will he responsible for the evaluation of the 
timeliness and utility of the' acreage, yield, and 
production information produced by the' AES (see 
section 3.^. 1.1, LACIE Project Plan, August 1975, 
LACIE-C00605). 

AGENCY ROLES 

USDA participation in 'the' IE function will include 
a full-time USDA LACIE IE staff '(USDA IE) and 
selected reviewer/ advisors from participating USDA 
user agencies and staffs. Details of NASA and NOAA 
participation have been proposed which encompass the 
roles and responsibilities specified herein (see 
section 3.1, 3. 3.2.2, and 3. 3. 2. 3)® This document 
will be revised to incorporate any changes in these 
roles and responsibilities jointly agreed to by 
NOAA, NASA, and the USDA. 
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SPECIFIC USDA IE RESPONSIBILITIES 


Within the general IE function, the USDA is respon- 
sible for the following (see section 3. 4. 1.2 of the 
LACIE Project Plan): 

A. Evaluating the timeliness and- utility of area, 
yield, and production information produced within 
the LACIE AES. 

B. For setting forth the' USDA policy as related to 
release of area, yield, and production information 
produced within the LACIE AES to all organizations 
or individuals not in direct support of the LACIE 
AES. 

C. For definition of the systems specifications 
necessary to support the design of an applications 
system within the USDA. 

D. For integrating information produced within the 
LACIE AES into ongoing operational activities asso- 
ciated with the appropriate functional agencies and 
services within the USDA. 

E. For performing benefit/cost analysis of acreage, 
yield, and production information produced within 
the LACIE AES. 
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F. For analysis and evaluation of the applicability 
of the LAC IE AES to the' derivation of area, yield, 
and production information pertaining to crops other 
than wheat . 

1.3 SCOPE 

While the USDA responsibilities detailed in section 
1.2 fall within the general information evaluation 
function, this Plan will be limited to the task, of 
evaluating the timeliness and utility of AES sched- 
uled and unscheduled reports as information for 
potential USDA user agencies and staff groups, and 
integrating AES wheat crop information into' ongoing 
USDA activies. 

The other USDA responsibilities detailed in 
Section 1.2 are being met as follows: 

USDA policy relating to release- of AES crop data 
has been established under Appendix I to the LACIE 
Management Guidelines and approved hy the Inter- 
agency Executive Steering Group (IESG) on November 5, 
1975. Benefit/cost analysis plans are inder devel- 
opment and are to be documented by March 15, 1976, 
according to the LACIE Schedule Level 1 dated 
October 1, 1975. Advanced system design will be a 
LACIE-Houston project for which, the USDA IE staff 
will provide support as requested. An analysis of 
the applicability of the AES to the' derivation of 
information on crops other than wheat will mainly 
be an outgrowth of efforts for the' separation of 
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confusion crops from wheat within the AES. Where 
appropriate in the evaluation of AES crop reports, 

IE may raise questions, request additional information, 
or offer suggestions to provide inputs related to 
these other USDA responsibilities. 

1.4 ORGANIZATIONAL STRUCTURE 

The evaluation of AES scheduled and unscheduled 
reports will be done by the IE reports evaluation 
team. This team will consist of NOAA and NASA 
participants, the USDA LACIE IE staff and selected . 
reviewer /advisors from the USDA Remote Sensing User 
Requirements . Task Force (RSURTF), and potential 
USDA user agencies. The report evaluation effort 
will he led by the USDA LACIE IE staff. 

1.5 OBJECTIVES 

With respect to AES reports evaluation, IE objec- 
tives are as follows: 

A. To evaluate the utility of AES scheduled and 
unscheduled crop reports where utility includes 
timeliness, continuity, objectivity, completeness, 
and accuracy — including precision and consistent 
differences and compatibility with USDA crop re- 
porting practices for such things as units of 
measure and levels of aggregation (see section 3 of 
the USDA User Requirements, dated October 1975). 

Bo To report the results .of these evaluations to 
Level 3 Management via Accuracy Assessment (AA) for 
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the purposes of (l) providing a measure of their 
potential value as information to the USDA, (2) sug- 
gesting improvements in formats frequency, accuracy 
or objectivity of AES reports, and (3) recommending 
changes related to the use of LACIE-type crop in- 
formation in ongoing USDA program activities, 

C. 'To provide USDA, NASA, and HOAA management with 
evaluation results for their use in monitoring 
LACIE progress toward fulfillment of the USDA User 
Requirements . 

Do To involve representative specialists from 
potential USDA user agencies in the report evalua- 
tion process for the purpose of developing user 
agency commentary on the potential utility of 
LACIE-type crop information in ongoing USDA program 
activities . 
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SECTION 2.0 
OVERALL APPROACH : 

IE will evaluate the reports produced by AES-CAS, 
both currently as they are received and again at 
the end of the growing season. These evaluations 
will be based on information received from. AES, 
other LACIE elements, USDA User Requirements, 
official USDA crop production estimates, .and such 
other information as foreign government crop reports, 
FAO reports, and trade news reports. NOAA and NASA 
will participate in the evaluation, and assistance 
will be obtained from USDA cooperating agencies and ■ 
the RSURTF. Detailed requirements of reports are 
shown in Section 4 of the CAS- IE ICD, revised 
December 1975 » LACIE C00709. 

The evaluations will be documented in scheduled and 
unscheduled reports called IE Monthly Reports (IMR) 
and IE Unscheduled Reports (IUR) (CAS-IE ICD, 

Section 5). These reports are directed to AA, to 
the LACIE Manager, -to the CAS Manager, to LACIE 
Level 2 Management, and to such USDA agencies as 
may require them.- Summaries of these reports may 
also be directed to Level 1 Management. 

2.1 DESCRIPTION OF TEE IE REPORT 

Each IE Report shall- contain at' least the following: 

1. A table of contents 

2. A summary of the report 

3. An introduction. 
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4. A copy- of the GMR checklist 

5. A list of acr.onyms and spe'cial terms 

6. - Statistical comparisons 
7 '. Comments 

8'. Requests, for additional data or for. changes 
in the CMR, as required 

2.1.1 Table of Contents 

- This will index the contents of the IMR/IUR down 
'to subsection level. 

2.1.2- ' Summary - ; 

This will be constructed so that it can serve as a 
stand-alone executive summary. It will both iden- 
tify the geographic and temporal area of the' CMR, 
and contain summary statements as to the relative 
timeliness, accuracy, precision, and general 
utility of the LAC IE estimates in the CMR or CUR. 

2.1.3 Introduction 

This will identify the CMR/CUR being evaluated as 
to area' of coverage,' as to dates of preparation and of 
receipt by 1-E, and as to the identity of the -review- 
teams , 

2.1.4 ' Checklist 

An annotated copy of the checklist submitted with 
the CMR will be enclosed here. 
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This 'will have a maximum of three parts.' Part 1 
will always he the list of acronyms and special 
terms as submitted by CAS with the CMR. Parts 2 
and 3 will be optional and will consist of terms 
to be added to or deleted from the' CAS list of 
acronyms and special terms in the future. 


2.1.6 Statistical Comparisons 


This section of the IE Report will include descriptions ■ 
of the following material. 


2. 1.6.1 USDA Estimates 

This will be a description of the USDA estimates used by 
IE for comparative purposes. 


2.1. 6. 2 Statistical Tests 

This will include a description of the statistical tests 
employed. 


2.1.6. 3 Evaluation 

This will include both tables of statistical comparisons 
and commentary to identify any significant differences 
■which may be present. 


2.1.7 Comments 

IE commentary on the LACIE Crop Report will deal with 
the putative utility to USDA, as defined in Section 2.2, 
of the- report, and possible explanations for any large 
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differences* 'between LACIE estimates and USDA values for 
the same item. 

2.1.8 Requests for Additional Data 

Any requests by IE for additional data to "be supplied by 
GAS, or for any changes in the CMR shall appear in this 
Seciton. 

i 

2,2 EVALUATION OF UTILITY 

The IE Evaluation Team will evaluate utility to potential 
USDA users. Factors- to be considered in evaluating 
utility will include timeliness and continuity, 
objectivity, completeness and accuracy of LACIE reports. 
These evaluations will be documented in the' IE reports . 

2.2.1 Timeliness 

IE will evaluate timeliness with respect to the 
following : 

2. 2. 1.1 AES adherence to the reporting schedule detailed in the 
CAS- IE Interface Control Document (ICD). 

2. 2. 1.2 Currency of spectral and meteorologlc data -used' in AES 
monthly reports as detailed in the CAS-IE ICD, Section 

4 . 1 0 

2. 2. 1.3 The’ potential impact of AES unscheduled reports on current 

LACIE crop estimates and/or USDA agency programs. 

2.2.2 Continuity 

IE checks for continuity will include the following: 
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2 . 2 . 2.1 


Does the existing AES reporting schedule provide 
adequate continuity? 


2. 2. 2. 2 Does AES comply with, the Schedule? 

2. 2. 2. 3 How do sources of crop, information currently used "by 
USDA compare with LAG IE for continuity? 

2.2.3 Objectivity 

IE will analyze and comment on input data quality 
problems or operating procedures which might impact 
on the objectivity of AES crop estimates, e.g., the 
use of FAS attache reports as a basis for the weekly 
weather summaries prepared by NOAA for use by AI 
personnel. 

2.2.4 Completeness 

To enable an evaluation of the utility of AES crop 
estimates , certain details are required in the 
scheduled crop reports (see CAS-IE ICD, Section 4). 
IE will examine AES crop reports to see: 


2„2o 4.1 That they contain: 

1. Crop estimates tables. 

2. Narrative. 

3. Element checklist 

4. Acronyms and terms. 

2. 2. 4. 2 That each section is completed according to the 
specifications in the CAS-IE ICD, Section 4. 
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2 . 2 . 4. 3 


That the narrative explains deviations from prescribed 
timeliness, format, completeness, and input data 
accuracy as detailed in the CAS-IE ICD, Section 4. 

2. 2. 4. 4 That the -narrative identifies and offers possible 

explanations for major deviations in area or yield 
estimates from, earlier reports or from a historic 
pattern. 

That the reports indicate the proportions of spectral- 
based data and historic-based data' used to' develop area 
estimates . 

That ' reports contain basic statistical descriptors such 

as standard deviations or standard errors, coefficients 

of variation, sample sizes, and the' like ■which pertain 

\ . 

to the AES estimates presented in the crop tables. 

That reports identify the’ biostages and calendar dates 
■which coincide with the acquisition of- the spectral 
and meteorological data represented by the respec- 
tive reports, 

2. 2. 4. 8 That reports are reasonably capable of being used as 
stand-alone documents 0 

2,2.5 Bell ability — Accuracy .and Precision 

IE will comment upon the accuracy and precision (as 
defined below) of the' LACIE estimates of area, yield 
and production. 


2 . 2 . 4. 5 


2 . 2 . 4. 6 


2 . 2 . 4,7 
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2 . 2 . 5.1 


Definitions 


Accuracy — How close does the estimate come to the 
value it is intended to measure? Also, in repeated 
trials, -what is the probability that the LAC IE estimate 
will consistently he above or below the target value? 

Precision — In repeated trials, how much scatter might 
one expect to find in the estimates. 

2.2. 5.2 Domestic Estimates 

LACIE estimates- of wheat area, yield and production 
for (portions of) the United States will be evaluated 
with respect to estimates for corresponding areas as 
published by the Statistical Reporting Service, USDA, 
and with respect to the accuracy requirements listed 
in the USDA User Requirements, Sections 2.4.2, 2.4.3, 
and 2.5 (also see Table A-I, page A-12, Appendix A of 
this document.) This will include an evaluation of the 
extent and of the significance of any differences. Pre- 
liminary LACIE estimates will be compared with USDA 
estimates made at the same stage of crop development as 
- a means of tracking differences. Then at the’ end of the 
season, the preliminary LACIE estimates, will be conqaared 
with USDA estimates made at the same stage of crop 
development as a means of tracking differences. Also at 
the end of the season, the preliminary LACIE estimates 
will be compared with the final USDA estimates. 
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2. 2. 5*3 Foreign Estimates 

The availability and objectivity of USDA or other 
independent estimates for foreign wheat area, yield 
and production has traditionally varied by country. 

Also, there is no current mandatory USDA release 
schedule for such estimates. 

Canada ; 

The LACIE early season estimate for spring wheat area 
in the indicator region will be compared with -the 
official Canadian June Intended Acreages Report for 
Project guidance only. This- will not represent a 
measure of LACIE accuracy. - 

The September' CMR on area estimate will -be' compared 
witli the Canadian- Preliminary ‘Estimate- of Acreages as 
an indication' of LACIE accuracy even though there is no 
basis for confidence limits on the data from 1 Canada. 

There: is no 'Canadian estimate for wheat yield -or pro- 
duction scheduled until the November report which is 
listed in the Official Field Crop Reporting Calendar 
as "date uncertain". Therefore, LACIE yield and pro- 
duction estimated for Canada ‘will be evaluated' on a 
subjective -basis. 

IE will also keep a- tracking ‘record of ' LACI'E estimates 
and USDA/FAS or other independent estimates for com- 
parison with a' January 1977 Canadian "final" area, yield 
and production report. 

USSR : 

LACIE estimates of area, yield, and production for the USSR 
winter and spring wheat indicator regions will be tracked 
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along with, any available independent estimates for the 
corresponding regions which might -become -available. 
Whenever an official estimate becomes available the 
progress of LACIE estimates and any independent estimates 
will be compared with the official estimates- as a 
measure of accuracy. 

In addition to the above tracking approach. Commodity/ 
country experts in FAS and ERS will be asked to furnish 
subjective opinions on the accuracy of the LACIE 
Canadian and USSR estimates after the initial evaluation 
by IE. These will be incorporated in followup IDE’s or 
the next succeeding IMR for that country. 

2.2.6 Compatibility . 

IE will compare AES reporting' systems with those -of 
the FAS, the Economic Research Service (ERS), and'SRS 
published crop estimates to determine general' com- 
patibility or harmony. This will include such items 
as units of measurement and level of aggregation.. IE 
may -also review future needs of USDA program, areas . 

2.3 INTEGRATION INTO ONGOING PROGRAMS 

IE will evaluate the’ methods for and the feasibility 
of integrating LACIE-developed estimates into appro- 
priate USDA user agencies. The operational approach 
will include involving .user agency representatives in 
the report evaluation process., 

2.4 ORGANIZATIONAL INTERFACES 
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CAS-IE Reporting Interface 

Hie Crop Assessment Subsystem (CAS) of AES will be the 
supplier of AES crop reports to IE. Hie CAS-IE ICD 
regulates this reporting interface. 


2. it. 2 


IE Evaluations of AES Crop Reports 


IE evaluations of AES crop reports service three .feedback 
loops . 


2. it. 2.1 Feedback to LACIE’ Level 3 Management 

This will provide the LACIE Manager, the AA, and the 
CAS Manager with a means of monitoring subsystem 
performance in comparison with existing reporting 
methods and identifying areas of possible 
improvement in the LACIE process. The CAS-IE ICD 
regulates this interface. 


2. it. 2. 2 Feedback to ROAA, MSA, and USDA Management 

This loop will be via the respective' Level 2 'Managers. 
It will provide the agencies with a means of monitoring 
LACIE performance. Since this essentially will be a 
staff function, there will be no interface control 
required. 


2. it. 2. 3 Feedback to USDA Agencies 'and Staffs 

This loop will facilitate interaction With USDA agencies 
and staffs in the evaluation process to provide commentary 
relating to USDA needs, to facilitate the integration of 
LACIE crop estimates into ongoing USDA program activities, 
and ultimately to increase their awareness and interest 
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Figure 2-1.- LAC IE information 'evaluation functional flow chart. 











in the LACIE product* Internal USDA procedures will 
govern this interface. 

2.5 FACILITIES TO BE USED 

IE sees no need for substantial facilities in addition 
to those already allotted. 

2.6 COORDINATION AND SCHEDULE TRACKING 

IE will maintain a detailed log of incoming and outgoing 
communications and reports containing LACIE data to 
provide eoordinaiton and schedule tracking with respect 
to the CAS-IE interface. 

2.7 CONSTRAINTS WHICH MAKE THIS APPROACH NECESSARY- 

t * 1 * » 

USDA crop production reporting systems and requirements 

are described in the USDA User Requirements document. 

Constraints include: 

A. Present and planned data processing capabilities 
of the USDA. 

B. Statutory requirements on the FAS and SRS. for 
estimating foreign and domestic crop production. 

C . USDA program area needs for reliable timely world 
crop estimates. 

2.8 FLOW DIAGRAM 

A simplified explanation of IE interfaces and operational 
flow is provided by the flow chart Cfig. 2-1 ) . A more 
detailed flow chart is shown in Section 1.2 of the CAS-IE 
I CD. 
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SECTION 3.0 

OPERATIONAL PROCEDURES 

This section presents the agency roles, resource 
needs, evaluation procedures, and reporting of re- 
suits of the evaluation of AES crop information. 

3.1 AGENCY ROLES 

3.1.1 ' USDA IE Staff 

The USDA IE staff will have the lead responsibility in 
IE and -will coordinate the interagency efforts. It 
will evaluate the utility of AES reports to potential 
USDA users and document the results of the evaluation 
and recommendations concerning the utility of AES crop 
reports as information. It will coordinate and track 
IE reporting to AA. 

3.1.2 NASA IE Member 

The NASA IE member will develop comments on AES crop 
’ reports with respect to wheat area inputs provided to 

CAS, and CAS's use of those inputs for their utility 
as information to users. 

3.1.3 NOAA IE Member 

The NOAA IE member will develop comments on AES crop 
reports with respect to wheat yield inputs, provided 
CAS, and CAS's use of those inputs for their utility 
as information to users. 

3.1.4 USDA User Agency and RSURTF Reviewers 

USDA User Agency and RSURTF representatives who have 
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■been designated as LAC IE crop report reviewers will 
cooperate with the USDA IE staff in the evaluation 
•process as agreed to within the USDA. 

3.2 RESOURCE REEDS 

The main resource needs for IE will he files of 
published and unpublished United States and foreign 
wheat crop estimates. The USDA IE staff, with 
assistance, from the Technical Support Group, will 
create and maintain these files in USDA LACIE Head- 
quarters offices. They will encompass at least one 
full crop year .and will consist of all pertinent SRS 

■ * i , 

and independent United States crop estimating service 
releases; all FAS, ERS, and non-USDA releases per- 
taining to foreign wheat estimates; and all unpub- 
lished FAS weekly foreign wheat estimates that were 
current as of the first of each month. Published 
and unpublished USDA estimates will be in the IE 
system by the day that the Maximum Protection 
Period empires -for the corresponding CMR. The IE 
staff will create and maintain a file of all LACIE 
crop reports , activity and management reports , and 
management control documents. The crop reports 
and activity and management- reports will be main- 
tained on a current basis with as many of the pre- 
ceding reports kept on file as can be obtained on 
a retroactive basis', ‘ A complete set of management 
control documents will be obtained. In addition, 
the' IE staff will build a file of technical refer- 
ence materialo 
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A partial listing of the resources to he maintained 
is shown, in Appendix B of this document.., | - ; • 

3.3 INFORMATION EVALUATION PROCEDURES AND REPORTS 

3.3.1 Coordination of Evaluation 

The USDA IE staff will have the lead responsibility 
for the informat ion evaluation' -team and will call 
and chair meetings, furnish meeting space, and copy, 
assemble, and distribute the combined evaluation 
reports. The staff will also have the lead 
responsibility 'for the preparation of the reports 
as shown in Sections 3. 3. 2.1 and 3. 3. 2. 3. 

NASA and NOAA will conduct evaluations of AES 
reports and contribute to the IE reports as 
described in Sections 3. 3. 2. 2 and 3. 3. 2. 3. These 
details are subject to agreement between the USDA 
IE staff and the respective agencies. 

Details of involving USDA user agencies and the 
RSURTF will be coordinated by the USDA LACIE 
Project Manager' or his designee. 

3.3.2 . Evaluation ' of AES Monthly ' Crop ‘Reports (CMR 1 s ) 

As -soon as the Maximum Protection Period for a 
report has expired, preliminary evaluation of the 
CMR will begin. Unless other arrangements have 
"been made, a copy of the crop report will he 
immediately sent by FAS messenger to NOAA and 
NASA IE team members for their preliminary evalu- 
ation. 


3-3 



3. 3.2.1 


Preliminary Evaluation by USDA IE , Staff * . 


The USDA IE staff will perform a preliminary- 
evaluation of the AES' monthly crop report -’( CMR) 
for its utility to potential USDA users according 
to the criteria described in Section 2.2. f They 
will document the results in a report' according 
to. the provisions- of Section 5.1 of . the CAS-IE 
IGD. • The' procedures for comparing LACIE crop 
estimates with current USDA estimates and a 
sample table of comparisons are shown in - 
Appendix -A » Each IE report will include at least 
the items cited in Section 2.1. 

3. 3. 2. 2 Preliminary Evaluation ~by NASA and’-NOAA 

Concurrent with the preliminary evaluation by- USDA 
NCAA and NASA will prepare independent preliminary 
evaluation comments on the report'.. These comments 
will concentrate bn, hut not he limited to, each 
agency's respective area of major responsibility 
within LACIE (see Sections 3.1.2 and 3.1.3}. As 
appropriate, each will address the general subject 
of episodic events capabilities . 

The -content of the NASA and NOAA evaluation will he 
incorporated into a joint report by the USDA IE staff. 

3. 3.2.3 -Final Evaluation of CMR 

At the discretion of the IE team, follow-up evaluation 
reports may he developed. These- will appear as 
Information Evaluation Unscheduled Reports (IUB).. These 
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may be prepared and submitted jointly or individually. 
(CAS-IE ICD, Sections 5.1 and 5*2) 

3.3.3 Evaluation of Other AES Reports 

AES reports i in addition to the CMR, include CAS 
unscheduled reports (CUB) and CAS country annual reports 
(CAR) — described in the' CAS-IE ICD, Sections k.2 and 
^.3. There are no prescribed evaluation procedures for 
these reports. CUR’s by their nature will vary in 
content. The evaluation of CAR’s will generally provide 
input to the project end of phase evaluations and will 
be guided by the requirements of those evaluations at 
the discretion of the IE team. 

3.3. ^ . ' Review by Potential USDA User Agencies and Staffs 

Review of LACIE crop reports and IE evaluations will be 
governed by internal USDA procedures agreed to by the 
appropriate USDA representatives. Appropriate comments 
and suggestions by USDA user agencies and staffs will be 
incorporated into an IUR. 
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SECTION U.O 

IMPLEMENTATION SCHEDULES FOR ASSIGNED TASKS 

IE will conform to the Reporting Schedule detailed 
in the CAS-IE ICD in performing its reports eval- 
uation function. Calendar dates are dictated by 
the following constraints: 

A. SRS release dates for Unites States crop re- 
ports. 

B„ Crop data protection requirements. 

C. Estimates of the minimum time required hy 
supporting LACIE systems and subsystems to perform 
their assigned tasks. 



SECTION .5.0 
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APPENDIX A 

IE EVALUATION PROCEDURES 
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SECTION Al.G 


OBJECTIVES 

IE will examine the AES crop estimates in order to: 

A. Determine If there are significant differences 
"between the AES estimate's) and the current USDA 
estimates of the same item. 

B. Determine if the relative standard errors are 
so large that LACIE estimates are in danger of 
exceeding the' USDA User Requirements for accuracy 
and precision when aggregated to the country level. 
These requirements are shown in tables 3-1 and 3-2 
of the USDA User Requirements 3 published October 
1975. 
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SECTION A2.0 


INDEPENDENT ESTIMATES 


The statistical comparisons made "by IE (see fig. 

A-l as an example) shall treat the current USDA 
estimates as though they -were without any sampling, 
forecast, or estimation error. This is because 
there is a definite problem in obtaining valid 
estimates of the sampling or estimation errors for 
either the United States or for other countries. 
However, IE will attempt to obtain, and to include 
in comments regarding utility of the estimates, 
approximations of the coefficient of variation of 
the differences between historic forecasts for the 
same date and the final estimates. 

Comparison’s with any available estimates from other 
reputable sources may be furnished' when the dif- 
ferences between the LACIE and USDA estimates are 
large » This, however, should not be taken to imply 
that either the LACIE or USDA estimate is more „ 
realistic. 
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A2.1 


IN-SEASON MONTHLY EVALUATIONS 


During the growing season, evaluations of LACIE 
estimates will "be "based on comparisons of current 
LACIE estimates to current USDA estimates for the 
same item. "Where appropriate, LACIE estimates will • 
also he compared with sources of foreign estimates 
used "by USDA in making the official USDA estimate. 
Figure A-l is an example of the form that will he 
completed for these comparisons. 

A2.1.1 Approach 

USDA User Requirements indicate that domestic LACIE 
■estimates for Phases II and III should achieve a 
reliability of at least 15 percent for the initial 
estimate and at least 10 percent for mid-season and 
later estimates (table A-l). The guidelines for 
foreign estimates are less stringent (see table A-Il). 

A2.1.2 USDA Estimates Used 

For the United States, the current published SRS 
estimate will he the USDA estimate compared with 
LACIE estimates for evaluations during the growing 
season. For other countries, the most recent FAS 
working estimates will be compared with LACIE 
foreign estimates. 
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POST-HARVEST AND FINAL EVALUATIONS 



A2.2.1 Approach 


LACIE estimates made throughout the growing season will 
be compared with- post-harvest and final USDA estimates 
to test compliance with the accuracy requirements shown 
in tables A-I and'A-II and' levied in' the' USDA User 
Requirements . 

A2.2.2 Estimates To 'Be Compared With LACIE 

The USDA estimate's of area, yield, and production of 
wheat for post-harvest evaluation of LACIE estimates 
for the United States will be the "final" SRS esti- 
mates for individual states as published in the 
December 1 Crop Production Report, to be issued about 
December 10.. However, preliminary evaluations may be 
performed using the SRS estimates for winter wheat as 
published in the' August 1 Crop Production Report, or 
for spring wheat as published in the October i Crop 
Production Report. Estimates of acreage, yield, and 
production at the county and crop reporting district 
level may also be used as and when published by the 
individual SRS state offices. 

For other countries, the LACIE monthly and final 
estimates will be compared with the "best" post- 
harvest estimates or area, production and yield 
known to USDA. 
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SECTION A3.0 


LACXE OUTPUT- STATISTICS. 


Details of output statistics required from. LACIE 
are shown, in the CAS-IE ICD, section 4. 
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SECTION A4.0 
EVALUATION 


The following "will he performed for each LACIE 
estimate which can "be matched with a USDA estimate 
of the same quantity. 

Ab.l SIGNIFICANT DIFFERENCES IN LEVEL OF ESTIMATE 

Any difference between the LACIE estimate and the 
independent estimate which has an absolute value 
larger than 1.645 standard errors of the LACIE 
estimate is to be flagged as a significantly large 
difference. Such a difference does not necessarily 
imply that the LACIE estimate is grossly in error. 
However, all such occurrences will be included 
in the evaluation report for possible action by the 
LACIE AES in Houston. 

A^. 2 STANDARD ERRORS OF THE ESTIMATE 

To ensure that the LACIE estimates aggregated to 
the national level meet the' criteria for accuracy 
as stated in tables A-I and A-II, it is only nec- 
essary to ensure that the relative standard errors - 
or coefficients of variation of the estimates 
for any particular subregion do not exceed some 
critical value. A set of optimally derived criti- 
cal values for the major crop reporting districts 
in the 9-state Great Plains region of the United 
States is given in table A-III. The details of 
the construction of table A-III are given in sec- 
tion 5 of this Appendix. Similar tables will be 
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constructed' for other countries for use hy IE, as 
appropriate, hut -will not he included in this 
document . 
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SECTION A5.0 


TECHNICAL NOTES 

5d ' CONSTRUCTION OF TABLE A-III 

Given that specified number (n^) of sample seg- 
ments have teen allocated to b different subregions 
in accordance with an optimum sampling plan, and 
-given that the total number (n) of sample segments 
allocated is such that estimates produced by the 
sample survey would have at least a specified level 
of accuracy, then a first approximation to the level 
of accuracy that should be expected for any sub- 
region i can he computed as: 

CVi = CV r /n r 7 where 

CV1 is the expected coefficient of variation for 
estimates from sub-region .i„ 

CV^ is the specified coefficient of variation 
(measure. of accuracy) for the region, 

is the total number of samples allocated to the 
region, and 

n 1 is the number of samples allocated to the i 
subregion. 

For the LACIE, the CV that would just meet the speci- 
fied levels of accuracy at the regional level can be 
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determined as follows , 'The U3M User Requirements for 
LACIE allow relative errors' as large as 10 or 15 percent 
for different types of estimates in different countries 
and at different times of the crop season. Interpreting 
these accuracy requirements in terms of a 90 percent 
probability statement leads to the conclusion that the 
largest permitted relative error for production should 
be divided by 1.645. Since the CV of the production 
estimate is computed as: 


it will be seen that CV . must be at least as large as 

prod 

CV or CV . , Further, 'if we assume that the area 
area yield. 

and yield estimates contribute equally to the variance 
of the production estimate, then we can compute optimum 
values as 


CV = CV ‘ . 
area yield 


~2 + /k + 4cv; 


prod. 


/2 


A-10 



A-ll 


TABLE A— I . - ACCURACY OF LACIE ESTIMATES FOR THE' UNITED STATES 
AS COMPARED WITH THE FINAL USDA ESTIMATE 3 


LACIE estimates 

Phase I 
(1975-1976) 
estimate, 
percent 

Phase II 
(1975-1976) 
estimate, 
percent 

Phase III 
(1976-1977.) 
estimate, 
percent 

Initial 

Planted area 

+20 

+15 

+15 

Yield 

+20 

’ +15 

+15 . 

Production 

. +20 . 

+15 

+15 

Mid-season 
Planted area 

+15 

+10 

+10 

Yield 

+15 

+10 

+10 • 

Production 

+15 

+10 

+10 ' 

Post-harvest • 
Harvested area 

+10 

+10 

+io • 

Yield 

+10 

+10 

• +10 • . 

Production 

+10 

+10 

+10 

Final 

Harvested acres . 

+10 

+10 

+10 

Yield 

+10 

+10 

+10 

Production 

+10 

• +10 

+10 . 

j 



' 


a Source: table 3.1 of USDA User Requirements, October 1975. 















TABLE A— II.- ACCURACY OF LAC IE ESTIMATES FOR OTHER COUNTRIES AS 
COMPARED WITH THE FINAL USDA ESTIMATES FOR EACH COUNTRY 9 


LACIE estimates 

Phase II 
estimate, 
.percent . 

Phase III 
estimate, 
percent 

Initial 



Planted area 

+20 

+15 

Yield 

+20 

+15 

Production 

+15 

+10 

Mid-stage 



Area 

+ 20 

+15 

Yield 

+ 20 

+15 

Production 

+ 15 

+10 

Pre-harvest 



Area 

+ 20. 

+ 15 

Yield 

+ 20 

+15 

Production 

+15 

+10 ’ 

Final 



Area 

+15 

+10 

Yield 

+15 

+10 

Production 

+10 

+ 10 





a Source: table 3.2 of USDA User Requirements, 

October 1975. 
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TABLE A— III . - OPTIMALLY DERIVED COEFFICIENTS OF VARIATION 
FOR LACIE , ESTIMATES OF WHEAT YIELD, AREA, AND 
PRODUCTION IN THE GREAT PLAINS STATES 


State or region 

Optimal level of accuracy 

lose 

15 % 

Area/yld. , 
percent 

Prod. , 
percent 

Area/yld. , 
percent ' 

Prod. , 
percent 


4.30 1 

6.08 

6.44 

9.12 

9 Great Plains States .... 

5.35 

7.57 

8.01 

11,35 

.5 Winter Wheat States .... 

6.99 

9.90 

10.48 

14.86 

Colorado 

19.01 

27.12 

28.21 

40.68 

Kansas 

11.80 

16.74 

17.62 

25.11 

Nebraska 

18.19 

25.93 

27.02 

38.90 

- . Oklahoma ............... 

17.03 

24.26 

25.33 

36.39 

Texas 

15.41 

.21,92 

22.95 

32.88 . 

‘2 Mixed Wheat States 

11.21 

15.91 

16.76 

23.86 

Montana 

13.94 

19.81 

20.79 

29.71 

South Dakota 

. 18.72 

26.71 

27.80 

40.06 

2 Spring Wheat States .... 

12.24 

- 17.37 

18.27 

26.06 

’’ - Minnesota 

• 29.46' 

42.55 

43.17 

. 63.83 

North Dakota 

13.40 

19.03 

19.99 

28.55 
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For CAS Monthly Report dated 

Retain under Restricted Access until 
Today's date 


Information Evaluation 'Comparisons of Standard US DA Crop Estimates and LACIE'Estfmates 


CAS Monthly Report No. 
USDA Publication:' 


Type of Wheat:' Winter { ) Est.: Area ( ) 

Spring ( ) ' Yield ( ) 

* ‘ Prod. ( ) 


Code 

l.D. 

USDA 

est. 

LAC IE estimates * 

Comparisons 

Value 

S.E., 

C.V. 

USDA/ 
LAC IE 

USDA 

LACIE 

dif. 

Dif./S.E. 

of 

LACIE 

>/ if ratio , 
exceeds 
1.645 

Max'. 

allowable 

C.V. 

V if c.v. 

not 

acceptable 



i 





• 

• 
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Figure A-l.- Information evaluation comparisons of standard USDA crop estimates 

and LAC IE estimates. 




APPENDIX b 

partial LISTING of IE 

IJIFQR14ATIOR RESOURCE FILES 



A. LACIE Management Control Documents 

1. Memo of Understanding, USDA-WOAA-KASA , 9/1/74. 

2. Memo of Understanding, FAS-ARS-ASCS-ERS-SCS-SRS. 

3. USDA Secretary* s Memorandum 1870, 4/8/75* 

4. Assistant Secretary Xeutter f s Memo to Agency 
Heads, 4/8/75. 

5. LACIE Project Plan. 

6 . LACIE Operations Plan 0 

7. LACIE Documentation Plan. 

8 . USDA User Requirements. 

9. LACIE Baseline Requirements Documents. 

10. LACIE Interface Control Documents, 

11. LACIE Implementation Plans'. 

12. USDA LACIE Administrative Guidelines. 

* j , 

13. Others. 

B, LACIE Reports 

1. CAS' Monthly Report's. 

2. CAS Unscheduled Reports „ 

3. CAS Country Annual Reports. 

4. IE Monthly Reports. 

5. IE Unscheduled Reports. 

6 . LACIE Management Reports. 

7 . LACIE Operations Bulletins (weekly ) 0 
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8, LACIE Activity Reports. 

P. Others. 

C. Foreign Crop Reports 

1. Agricultural Attache Reports (Grains) {mostly 
quarterly) 0 

2. Agricultural Attache Telegrams (TOFAS) on Grains. 

3. U.S. Embassy Cables (Grains). 

if. FAS Circulars (Grains) (Unscheduled). 

5. FAS M Series Publications (Grains) (Unscheduled). 

6. Foreign Agriculture Magazine (weekly). 

7. FAS World Agricultural Production and Trade 
(monthly) . 

8„ FAS Unpublished Wheat Crop Estimates (weekly).. 

9. Department of State Airgrams. 

10. FAS Weekly Highlights Report, 

. 11. USDA Interagency Commodity Estimates Committee 
Report (Wheat ) , 

12. ERS Weekly Highlights Report. 

13. ERS Wheat Situation Report. 

14. Reuters Hews Service Grain Market Report. 

15. London Public Ledger. 

16. TOEPFER Report (Hamburg). 

17. Canadian (DBS) Wheat Crop Reports. 

18. FAO Reports. 

19* US-USSR Secretariat Reports. 
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D. UiS. Wheat Crop Reports 

1. SRS December Crop Production. 

2. SRS Seeded Acreage and Indicated Production 
of Winter Wheat - December. 

3. SRS Annual Summary: Acreage, Yield, and Production. 

b t SRS' January 1 .Prospective Plantings - Spring Wheat. 

5. SRS March Prospective Plantings - Durum Wheat. 

6. SRS May Crop Production. 

7. SRS, June Crop' Production. 

8. SRS June Acreage. 

9= SRS July Crop Production. 

10. SRS August Crop Production. 

11. -SRS September Crop Production. 

12. SRS October Crop Production. 

13. SRS County Estimates. 

l4 = SRS Selected State Office Releases Showing Crop 
Reporting District Data. 

15= SRS Weekly Weather Crop Bulletin. 

l6= United States Agricultural Census. 


nasa-jsc 
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